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ABSTRACT 



Carnegie Mellon Online has been used to deliver several 
university courses to thousands of students at Carnegie Mellon University 
(Pittsburgh, Pennsylvania) over the last 2 years. This paper describes the 
experiences gained from redesigning an introductory art history course using 
Carnegie Mellon Online to deliver supplemental course content. The 
technology, course content and structure, assessment tools, and course 
management capabilities are discussed, as well as lessons learned from 
creating this course. Plans for the next version are also highlighted. 
(Author/AEF) 
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This paper describes the experiences gained from redesigning an introductory art history course using 
Carnegie Mellon Online to deliver supplemental course content. Carnegie Mellon Online is a database driven, 
web-based educational tool that has been used to deliver several university courses to thousands of students at 
Carnegie Mellon University. The technology, course content, and structure, assessment tools, and course 
management capabilities are discussed, as well as lessons learned from creating this course, and plans for the 
next version. Descriptions of Carnegie Mellon Online courses are available at 
http : //online . web . emu. edu/. 



The Technology: Carnegie Mellon Online 

Carnegie Mellon Online is a unique, database driven, educational tool taking advantage of the World 
Wide Web to provide student-centered instruction. The system generates customized content, such as assess- 
ments and feedback for each student and tracks the student through a course while maintaining course- 
specific rules and policies. Using this system, the instructor sets guidelines and prerequisites for student 
advancement through the course material, allowing the progress of each student to be individualized and 
paced for his or her learning style. It also provides students with more control over their learning experience 
by permitting them to access their class at any time they choose and as often as they choose, as best suits their 
needs. 

Carnegie Mellon Online is not limited to a specific discipline or type of course. Courses in all 
university disciplines can make use of its unique, student-centered approach to education in a variety of ways. 
It can be used to offer completely online courses that never meet in a classroom, can act as an online 
management and learning tool to complement classroom-based courses, or it can act solely as an assessment 
tool. 

One of Carnegie Mellon Online’s strengths is it’s ability to handle assessments. Like the rest of the 
system, assessments are data-driven; a standard set of code generates individualized assessment descriptions, 
generates the HTML forms, delivers the assessments, accepts the assessments, automatically grades certain 
types of assessments, and delivers feedback to the students. 

Security is always a concern when delivering classes via the web. A student must use a password to 
access any course learning material, assessments, or feedback. Students use their university password to 
access Carnegie Mellon Online. This same password is used to access their email, grades, course registration 
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sites, etc., which reduces the likelihood that students will share their passwords with other students to cheat on 
their course work. 

The Course: Pre-Industrial Visual Cultures; to 1789 

Pre-Industrial Visual Cultures; to 1789 (PVC) is the first course in a three-semester sequence of 
courses required of majors in the School of Art at Carnegie Mellon University. It is geographically and 
chronologically the most broad of the three courses, introducing students to the ideas and artifacts of world 
cultures from Paleolithic cave paintings to the French Revolution and including Africa, India, China, Japan 
and the Ancient Americas. 

The structure of the course is twice-a-week lectures and once-a-week discussion sessions. Lectures 
focus on works of art in their cultural context. For example, a lecture on the art of India investigates how 
visual expressions reflect historical background, politics, geography, social structure, religious beliefs, and the 
role of the artist in India. One class per week focuses on synthesis topics, such as mythology, the role of the 
artist, and patronage, which explore thematic comparisons across cultures. In these sessions, students are 
assigned to work in small groups to brainstorm ideas, discuss issues and make summary presentations. For 
example, when we discussed archaeology as a topic, each student group developed a set of questions that an 
archaeologist would ask about an artifact and made conclusions about the artifact that they were given. The 
assignment for the topic required that students discuss what an archaeologist in the future would say about our 
culture from unearthing a shopping mall or a gallery showing their own artwork. 

Since the Spring of 1995, Carnegie Mellon art historian Mary Schmidt has taught this course by 
projecting images and text in the classroom with a custom-designed, infinitely modifiable digital image 
database. She has also developed a text-based web syllabus with course overview, calendar, readings, links to 
relevant websites, and essay assignments. 

PVC Online 

In the Summer of 1997, Schmidt, Rehak, Blackmon, Bajzek, and a team of others collaborated to 
turn the classroom teaching tools and web syllabus into comprehensive web-based learning tools via Carnegie 
Mellon Online. The result was the delivery of a database driven, interactive website that contains all of the 
materials supplementary to the classroom experience - except for the required text book and visits to the 
museum. The site contains a course overview, calendar, images, timelines, maps, reading assignments, and 
links to other websites. It quizzes students on their reading assignment before each class period and accepts 
their weekly, electronically submitted essays. On a daily basis, it gives them their personal status in the course 
with regard to upcoming and/or overdue assignments and their cumulative grade. When students login, it 
opens to the current topic for the day. 

The first iteration for Spring 1998 exceeded some aspects of our original concept and fell short on 
some others. The assessment component proved to be a powerful tool. Students developed the sound practice 
of keeping up with the readings, and they improved their test taking abilities over the semester. They were 
prepared for class! No weekly essays for this class of 55 students were lost. No my-dog-ate-it excuses. 
Admittedly there were computer glitches; many students required extra help to set up their web browsers to 
upload their written assignments. 

We planned a searchable database of images but due to time constraints, settled for a sample of 
linear sequence images for each culture topic. Likewise, we planned a glossary but temporarily postponed its 
development. Both components are priorities for the second iteration for Spring 1999. 
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Structure of the Online Course 



Content 

The PVC Online interface structure gives students access to guideline information about the course 
and to their status with regard to grades and assignments as well as comprehensive materials for each day’s 
topic. 

Global Content 

Instructor information lists when and where the course meets, and gives the instructor’s phone 
number, email address and office location. A course description outlines course goals, policies, and proce- 
dures and instructs students how to navigate the site. A calendar and a topics list display the schedule for the 
semester in two different formats. A status chart gives students individualized reports of the entire semester’s 
quizzes and assignments - what has been completed and when each was submitted, what’s late, what’s due 
next, and what the cumulative score to date is (Figure 1). Another page lets each student know what’s next, 
what is overdue and what the deadline is for each overdue item. 

Culture Content 

For each culture topic there is a brief introduction to the topic, reading assignments, map, timeline, 
images, links to other websites, and a quiz. Students may also, at any time, access global information for the 
entire semester. For example, students may toggle between the global map and local map; likewise between 
the timeline for the semester and the culture timeline, and they may read semester guidelines for quizzes, 
reading assignments, and links to other relevant websites. Images are selected examples for which the 
instructor holds the copyright. As already discussed above, four randomly generated questions comprise the 
quiz to be completed before class begins. 

Synthesis Content 

For each synthesis session there are a brief introduction to the discussion topic for the day, a reading 
assignment, and a description of the written assignment to be electronically submitted the following week. At 
the end of the assignment description are instructions for submitting an assignment electronically. 

Interface 

Students navigate the PVC Online website through clickable buttons. The interface is designed with 
buttons across the top for administrative information and left/right navigation backwards and forwards 
through the semester’s topics (Figure 2). When students log in to the course, the first page that is presented is 
for the cultural or synthesis topic that will be discussed at the next class meeting. In the left column are bi- 
colored buttons for course content materials. All of the information that students call up through navigation of 
the site appear in the central frame of the window. Only links to other websites call up new windows. Details 
follow. 

Header Row. 

Buttons in the upper left of the header include Course Description, Mary Schmidt (for who, what, 
when, and where information). Calendar and Topics List. In the upper right are Your Status, Do/Due Now, 
Today and Logout. The upper center frame indicates the current topic title with left and right arrows to move 
forward or backward through the topics for the semester. 

Left Column. 

There are two sets of bi-colored buttons, one each for culture and synthesis topics. Culture buttons 
include Home, Readings, URLs, Maps, Timelines and Quizzes. The light blue left side brings up semester- 
wide information; the dark blue right side, information specific to each day’s topic. 
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Assessments 

Because of Carnegie Mellon Online’s assessment capabilities, we decided to encourage students to 
be prepared for class by requiring them to take a short quiz based on the text before attending class. Like 
paper- based quizzes, the instructor had to decide what kind of quiz was appropriate for testing the students’ 
comprehension; she chose to use a four question, multiple-choice quiz. 

For paper-based quizzes, the instructor would simply write four questions per topic and photocopy 
the quiz for each student. 

However, since the Online technology can generate individualized, randomly generated quizzes for 
each student, the instructor divided each quiz into four topics and created three multiple-choice questions for 
each topic. For each student, the system chose one question from each of the four topics, randomly permuted 
the answers, and generated the HTML form for the quiz. 

In addition to creating the quiz content, the instructor had to determine the policy for the quizzes, 

such as: 

• when can a student start a quiz? when is the last time that a student can start a quiz? turn 
it in? 

• when is a quiz due? what is the penalty for turning it in late? 

• can a student choose to retake a quiz? how many times? is it the same quiz or another 
randomly generated quiz? what is the penalty? 

• can a student see any of the course’s online learning material during a quiz? 

• what feedback does the student get? 

In order to show the flexibility of Carnegie Mellon Online, the instructor’s policy decisions and 
rationale are presented. 

A quiz is due thirty minutes before the class meeting and may be started up to a week before the 
class. This discourages students from finishing a quiz just as the class starts. Late quizzes are penalized 25%, 
which is the equivalent of missing one question. A quiz may be taken up to two weeks after the class, which 
prevents students from waiting until the end of the semester to do all the quizzes. 

Students who do poorly on a quiz can elect to take a second randomly generated quiz, which carries 
a 25% penalty. Students are allowed to see all of the course’s online material during a quiz. 

After submitting a quiz, a student is not shown the correct answers or even the questions. The student 
is only told how many questions were answered correctly. This policy discourages students from compiling 
printouts of questions and answers and sharing them with current or future students. The instructor however 
has access to the individual questions and the student’s answers for each quiz. Students may discuss 
individual quiz results privately with the instructor, but have not expressed much need to exercise this option. 

The instructor realizes that these policies do not prevent students from working together on quizzes. 
However, the security system discourages students from giving their password to one student to take quizzes 
for the group, and the individualized quizzes make it harder for students to copy other students answers. We 
believe that these measures bring about the intended goal of these quizzes - to encourage students to read, 
think, and talk about the material before coming to class, whether they take the quizzes individually or in 
groups. 

Policy for Written Assignments 

Students are also required to submit one written assignment per week; this is done as a file upload 
through the course’s web interface. As with the quizzes, the instructor had to determine the policy for these 
written assessments. 

Written assignments are due on a day when the class does not meet. Late assignments are penalized 
25%. The assignment submission page is available from 30 minutes after the class in which the assignment is 



discussed until two weeks after the due date. This prevents students from submitting all assignments at the 
end of the semester. However at any point during the semester, the student may see the instructions for the 
assignment. 

Papers may not be rewritten and resubmitted. The system only allows a student to submit a single 
paper. Students are allowed to see all of the course’s online material while working on an assignment. 

After a student submits an assignment, an acknowledgment is given that the paper has been accepted. 

Students receive written feedback on their papers from the instructor. The instructor’s interface is 
described below. 

Management Tools 

Through a separate administrative interface, the instructor has electronic access to statistical 
information about quizzes, to individual student’s status reports, to grade summaries for the entire class, and 
to screens which allow changes and corrections to accommodate idiosyncrasies - student illness, adding and 
dropping the class, re-writing essays. 

For the first time, the instructor is able to do something she has always wanted to do, but didn’t have 
the time to do: compile a statistical analysis of students’ answers to quiz questions. The quiz summary 
displays in graphic format how many students scored 4 out of 4, 3 out of 4 and so forth; and displays how 
many right and wrong answers there are for each question. It gives her the opportunity to assess the clarity 
and fairness of questions as well as students’ understanding of the material. 

Since Carnegie Mellon Online compiles and reports each student’s cumulative numerical score and 
percentage, reporting mid-semester and final grades for 55 students is much less labor intensive than in the 
past. 

The instructor can change an individual student’s deadline for quizzes and assignments in case of ill- 
ness, can change a quiz score after a well-argued rationale for a wrong answer, can change a grade for a re- 
written essay, and can add or delete students on the class roster. 

Lessons Learned and Future Plans 
Rethinking assessments. 

Much of the time spent in moving the course to the Carnegie Mellon Online was spent in rethinking 
assessment tools. In previous years, the instructor would use less frequent tests based upon both the lectures 
and readings. With the new automated system, the instructor chose to use quizzes before class meetings to 
promote reading. But this brought a new problem. Test questions from lectures and readings which address 
issues of comparison across time and cultures are missing. The next iteration of the course will include 
monitored web-delivered mid-semester and final tests. 

Student Reaction 

Spring 1998 students are excited to be a part of this project. They showed we’re-all-in-this- together 
enthusiasm, exhibiting patience when there are computer glitches, studying together, and discussing art 
history. Other faculty who teach the same students relate that students think it is great to have “all this stuff’ 
available to them. Though these students have been thoroughly assessed through twice-a-week quizzes and 
weekly essays, there have been no complaints about the work load. 

A Priori Content 

In this first version of Carnegie Mellon Online, all content had to be entered before the beginning of 
the semester. This placed a large burden on the instructor to plan the course content, assessments, and policy 
well before the semester started. For this art history course, the course interface and policy were set about six 
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months before the semester; the majority of the content and assessment questions were in place about four 
weeks before the semester, with the remainder of the time spent refining the content and testing the system. 

On the other hand, the benefit of this requirement is that the instructor is now free to spend more 
time with the students and planning individual lectures. Furthermore, this allows students to see the content 
for the entire semester at any time during the course. It also prevents students from having an inconsistent 
experience with the course - namely the threat of a student seeing material, and then having it be changed by 
the instructor without the student’s knowledge, and the student not seeing the new material. 

Though the benefit outweighs the burden, many instructors may not want to commit so much 
planning time before a class and may not have the time to prepare the entire content before the class starts. 
The next version of Carnegie Mellon Online will allow an instructor to add and update the course’s content 
during the semester. 

Future Plans. 

The next version of this course will have several enhancements including a glossary, assignments 
based upon the web materials, and web-delivered mid-term and final exams with new kinds of test items in 
addition to multiple choice questions, such as matching, short answer, and image identification items. 

The most important addition will be an image database to replace the current linear sequence of 
sample images. Schmidt has over 2,000 digital images of artifacts relevant to this course. We will create an 
interface that will allow students to search the database based on several descriptors of the artifacts, including 
artist, location, period, style, theme, and type of image. The students will then have instant access to the 
found images. One of the exciting aspects of this database is that the term “image” is not limited to a single 
static image; instead, it includes any of the media that can be delivered over the web including movies, three- 
dimensional models, and virtual reality models. 

Conclusion 

For all of the participants in this project, the journey has been an intensive collaboration to integrate 
pedagogy and technology for the creation of a comprehensive and effective online teaching and learning tool. 
We are enthusiastic about the results and are planning improvements for the future. 
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Figure 1: A Sample Status Page for a Student, Showing Aggregate Performance As Well As Individual 
Grades. 
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Figure 2: The Timeline for the Japanese Cultural Unit. 
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